Method for processing images having specularities and corresponding computer program product

ABSTRACT

A method for processing images of a scene including a surface made of a material of unknown reflectance comprises the following steps: from at least 3 different positions of an image sensor, the positions and corresponding orientations of the sensor known, acquiring images of the scene illuminated by a light source, each image containing specularity generated by reflection of the light source from the surface and depending on the position, the shape and the intensity of the light source and the reflectance of the material; in each image, detecting each specularity and for each specularity, estimating a conic approximating the specularity. It comprises constructing a quadric representing the position, intensity and shape of the light source and the reflectance of the material, on the basis of the conics, and of the position and orientation of the image sensor during the acquisition of the images containing the specularities respectively approximated by these conics.

The field of the invention is that of the processing of imagescontaining specularities.

Reflection of light is observed when visible electromagnetic wavesencounter a surface that does not absorb all of their radiative energyand repulses some thereof. When the imperfections of a surface aresmaller than the wavelength of the incident light (this is the case of amirror, for example), all of the light is reflected specularly (i.e. theangle of reflection of the light is equal to its angle of incidence).This effect causes, in images, specular spots, i.e. visible elements, onmany materials.

In general, these specularities are not taken into account inmachine-vision algorithms because these algorithms are highly dependenton viewpoint, on the materials present in the scene, on the settings ofthe video camera (exposure time, aperture) and on the geometry of thescene. These specularities are also dependent on the shape of the lightsource (bulb, fluorescent tube).

Now, these specularities are of major interest in various contexts. Foraugmented reality (AR) applications such as sales aids, the insertion ofvirtual elements into a real scene must be as natural as possible inorder to allow an optimal user experience. Specifically, this insertionmust be stable and include optical artefacts (the luminous context ofthe scene) such as specularities and shadows, elements that areessential for a realistic appearance.

Moreover, regarding the problem of real-time video-camera localization,current algorithms may be disrupted by specularities. Specifically,these algorithms are based on temporal tracking of primitives (generallypoints of interest). The latter may be completely or partially occultedby specularities. Current methods limit, to a certain extent, theproblem by using robust estimation algorithms, which then consider thesezones of the image to be noise. Nevertheless, for certain viewpoints,these specularities may saturate the video camera and cause thelocalization algorithms to fail. Ideally, these specularities should beconsidered as primitives in their own right, able to provide a greatdeal of additional information for reinforcing these localizationalgorithms.

These optical artefacts may also play an important role in theunderstanding and modelling of the behavior of light in a scene.Specifically, it is possible to deduce from these specularities thegeometry of a surface on which they occur. These specularities may thusbe used for quality-control applications in order to verify theintegrity of a 3D surface during the manufacture of industrial parts.

The prediction of these optical artefacts and in particular theprediction of specularities is of major interest in the aforementionedapplications. However, it is a difficult problem because specularitiesdepend on the viewpoint of the observer, on the position, shape andintensity of the primary light sources and on the reflectance propertiesof the materials on which they occur.

The state-of-the-art regarding the estimation of light on the basis ofimages may be divided into two categories: the modelling of overallillumination and the reconstruction of primary sources.

Light is an essential element to the formation of an image.Specifically, the latter is the result of the interaction between lightsources and objects of various materials for a given sensor (eye, videocamera). This light is emitted by one or more light sources, which maybe categorized into two categories:

-   -   Primary sources corresponding to bodies that produce the light        that they emit. This category includes bodies having a very high        temperature such as the sun, flames, incandescent embers or even        a filament of an incandescent lamp.    -   Secondary or scattering sources corresponding to bodies that do        not produce light but that redirect received light. Scattering        is an effect in which a body, associated with a material, having        received light, partially or completely redirects it in every        direction. The amount of light scattered depends on the        properties of the materials of the objects receiving the light.        A scattering object is therefore a light source only when it is        itself illuminated by a primary source or by another scattering        object.

In their approach to modelling overall illumination, Jacknik et. al.propose in the publication “Real-Time Surface Light-field Capture forAugmentation of Planar Specular Surface” ISMAR 2012, an indirectreconstruction of overall illumination, taking the form of a map of theluminous environment generated on the basis of all the images of a videosequence. This reconstruction is used to achieve a realisticphoto-finish after a phase of initialization on a planar surface made ofa specular material. However, this method is limited because it makes nodistinction between primary and secondary sources. Therefore, thismethod does not allow specularities from unknown viewpoints to bepredicted.

The approach to estimating overall illumination of Meilland et. al.described in the publication “3D High Dynamic Range Dense Visual SLAMand its Application to Real-Time Object Re-Lighting” ISMAR 2013,presents a reconstruction of a primary (point) source achieved bydirectly filming light in the scene. However, this method is unsuitablefor more complex types of lighting such as fluorescent tubes representedby a set of point sources. In addition, dynamic light (turned on, turnedoff) cannot be managed and, with regard to specularities, materials arenot taken into account. Therefore, this method does not allowspecularities from unknown viewpoints to be predicted.

The reconstruction of primary sources according to the method of Laggeret. al. described in the publication “Using Specularities to RecoverMultiple Light Sources in the Presence of Texture” ICPR 2006, presents areconstruction of the direction of the primary source on the basis ofspecularities on a moving object observed from a stationary viewpoint.This application is limited; specifically, as specularities depend onviewpoint, it is necessary to re-estimate them in each position. Inaddition, neither the position, nor the shape of the light source areestimated and material is not taken into account. Therefore, this methoddoes not allow specularities from unknown viewpoints to be predicted.

The approach of Boom et. al. described in the publication “point LightSource Estimation based on Scenes Recorded by a RGB-D camera” BMVC 2013,details a method for estimating a primary light source that isconsidered to be point-like on the basis of a Lam bertian(nonreflective) surface, using a RGB-D (Red Green Blue-Depth) sensor.This approach only uses the diffuse component to estimate the pointsource, a synthesized appearance being compared with the actual scene inorder to approach the actual scene as best as possible. However, thismethod is not suitable for real-time application, and cannot handlefluorescent tubes but only point sources, nor can it manage the presenceof specularities (assumption of Lambertian surfaces). In addition, thismethod is limited to one light source and its shape is not estimated;furthermore, the specular component of materials is not taken intoaccount. This method does not allow specularities from unknownviewpoints to be predicted.

Regarding the prediction of specularities, there is no known method.

Therefore, there remains to this day a need for a method for processingimages containing specularities that optionally allows specularitiesfrom other viewpoints to be predicted.

The method for processing images including specularities according tothe invention is based on the construction of a new model combininginformation on the light sources (intensity, position and shape), on thereflectance of the materials on which the specularities form, andpossibly the roughness of the surface. From a technical point of view, alight source is reconstructed in the form of a generic 3D shape, here aquadric (sphere, ellipsoid, cylinder or even point). This empiricallight-source model, which is symmetric with the real light source withrespect to the surface, is reconstructed on the basis of imagescontaining specularities that are observed on a planar surface of agiven material, these images being obtained for given video-camerasituations (or more generally given positions of an image sensor). Usinga real-time detector of specularities and a video-camera localizationmethod, the model is reconstructed in the form of a quadric that is ableto represent not only a light bulb but also a fluorescent tube.

The constructed model allows a specularity in images, obtained from anew viewpoint, of the scene and including a surface made of thismaterial to be predicted. Specifically, this model includes theposition, intensity and shape of the light source and the reflectance ofthe material and possibly the roughness of the surface on which thespecularity is present. The zone predicted for the specularitycorresponds to a conic obtained by projection of the reconstructedquadric, for the given new video-camera situation.

More precisely, one subject of the invention is a method for processingimages of a scene including a surface made of a material of unknownreflectance, which comprises the following steps:

from at least 3 known different orientations and positions of an imagesensor, acquiring images of the scene illuminated by a light source,each image containing at least one specularity generated by reflectionof the light source from said surface and depending on the position, onthe shape, and on the intensity of the light source and on thereflectance of the material; and

in each image, detecting each specularity.

It is mainly characterized in that it furthermore comprises thefollowing steps:

for each specularity, estimating a conic approximating said specularity;and

constructing a quadric representing the position, the intensity and theshape of the light source and the reflectance of the material, on thebasis of the conics, and of the position and orientation of the imagesensor during the acquisition of the images containing the specularitiesrespectively approximated by these conics.

This method allows:

-   -   conjointly, information on the light sources and the properties        of the materials of the scene to be included;    -   specular zones to be predicted for new viewpoints, for a given        material;    -   a plurality of kinds of light to be managed (light bulbs and        fluorescent strips);    -   a plurality of light sources to be estimated simultaneously;    -   the model, in quadric form, to be constantly refined with new        viewpoints;    -   additional information to be given to the video-camera        localization method;    -   the state of the light sources (turned on, turned off) to be        updated;    -   any prior learning phase to be eliminated; and    -   operation to be in real-time on-CPU.

Finally, a model is constructed combining information on the lightsource and the materials of the scene and that also allows specularzones to be predicted in real-time and precisely.

When the scene is illuminated by at least one other light source andeach image contains at least one specularity generated by reflection ofeach other light source from said surface, the method furthermorecomprises for each specularity:

temporal tracking of the specularity and

matching said specularity, and therefore the conic approximating it,with a light source,

and the step of constructing a quadric is carried out for each lightsource, on the basis of the conics matched with said light source, andof the position and orientation of the image sensor during theacquisition of the images containing the specularities respectivelyapproximated by these conics.

It preferably furthermore includes the following steps:

for each specularity, minimizing the distance between the specularityand a conic resulting from projecting the quadric onto the imagecontaining said specularity along an axis determined by the position andorientation of the sensor, in order to determine a new quadric; and

iterating the preceding step until a preset stop criterion is reached,the new quadric resulting from the last iteration being a refinedquadric representing the position, the intensity and the shape of thelight source and the reflectance of the material.

It advantageously includes a step of choosing key images from theacquired images, depending on a predefined criterion of distribution ofviewpoints of the sensor in the scene.

When a light source is a point light source, the corresponding quadricis a sphere; and, when it is an extended light source, the correspondingquadric is an ellipsoid.

It may also include a step of calculating a prediction of the positionand shape of a conic (called the predicted conic) approximating aspecularity formed on a surface made of a material of known normal,depending on a new position and a new orientation of said image sensorduring the acquisition of a new image and on the projection of thequadric onto said new image.

When the material of the predicted specularity is the same as thematerial associated with the quadric, the size of the specularity isalso predicted.

Scales of projection of the quadric being preset, the conic predictioncalculation is advantageously carried out for each scale so as to obtainas many conics as scales, each conic corresponding to an intensity levelof the predicted specularity.

According to one feature of the invention, it includes a step ofcorrecting error in the position and orientation of the image sensor onthe basis of a calculation of a discrepancy between a specularitypresent in the image associated with said position and orientation and aconic resulting from the projection of the quadric onto said image.

According to another feature of the invention, the different positionsand orientations of the sensor are obtained using the same moving sensoror using a plurality of stationary sensors.

Another subject of the invention is a computer-program product, saidcomputer program comprising code instructions allowing, on the basis of:

at least 3 known different orientations and positions of an imagesensor, and

images of the scene illuminated by a light source, which images areacquired by said image sensor, each image containing at least onespecularity generated by reflection of the light source from saidsurface and dependent on the position, on the shape, and on theintensity of the light source and on the reflectance of the material,

the following steps to be carried out when said program is executed on acomputer:

-   -   in each image, detecting each specularity;    -   for each specularity, estimating a conic approximating said        specularity; and    -   constructing a quadric representing the position, the intensity        and the shape of the light source and the reflectance of the        material, on the basis of the conics, and of the position and        orientation of the image sensor during the acquisition of the        images containing the specularities respectively approximated by        these conics.

Other features and advantages of the invention will become apparent onreading the following detailed description, which is given by way ofnonlimiting example and with reference to the appended drawings, inwhich:

FIG. 1 shows a flowchart of the main steps of the method according tothe invention;

FIG. 2 schematically shows an example of specularity on a planar surfacefor an incident ray using the Phong model;

FIG. 3 illustrates an example of texture and specularity in two imagestaken from two different viewpoints;

FIG. 4 illustrates two different approximations of a specularity by anellipse;

FIG. 5 schematically shows the links between a light source and a planarsurface presenting specularities, and the corresponding virtual modelaccording to the invention;

FIGS. 6a and 6b schematically show a point light source (left-handimage) and an extended source represented by a set of point sources on asegment generating on a planar surface specular spots circumscribed byan ellipse (right-hand image);

FIGS. 7a and 7b schematically show two examples of distances between aspecularity and a corresponding ellipse, with the point/point distance(FIG. 7a ) and the Jaccard distance (FIG. 7b );

FIG. 8 illustrates a plurality of intensity levels of a specularity,respectively represented by conics; and

FIG. 9 schematically shows a plurality of concentric quadricsrespectively corresponding to a plurality of scales, each projectedquadric being a conic associated with one intensity level of thepredicted specularity.

From one figure to the next, elements that are the same have beenreferenced with the same references.

In the rest of the description, the expressions “top” and “bottom” areused with reference to the orientation of the described figures. Insofaras the image sensor may be positioned with other orientations, thedirectional terminology is given by way of illustration and isnonlimiting.

Knowledge of primary sources is essential for a better comprehension ofthe scene. To find these primary sources, a plurality of opticalartefacts, such as shadows and specularities, that are available in theimages, may be used.

1. Shadows correspond to zones occulted by an opaque object in a scene.These elements depend on the size of the occulting object and on itsdistance with respect to the primary sources. In addition, they aregreatly influenced by the shape of the latter.

2. Specularities correspond to a total reflection of the primary sourcefrom a reflecting surface. These artefacts are dependent on theviewpoint and the materials on which they are observed.

A light sensor (eye, video camera) is sensitive to these opticalartefacts. It is moreover the latter that generally assist a human beingwith interpretation of a scene (in particular information on depth, thecoherence of the scene, etc.).

For machine-vision applications, light is therefore an essentialelement. Knowledge thereof may be advantageous in various applications:

-   -   For augmented-reality applications, realism is greatly dependent        on the virtual insertion of these optical artefacts.    -   The algorithms used in the field of video-camera localization        may be disrupted by the presence of specularities. Knowledge of        these elements could improve these algorithms.    -   For the verification of properties of an object (geometry,        materials) in the field of industrial inspection.

The method according to the invention is based on a method forconstructing a virtual model allowing light sources of various shapes(light bulbs, fluorescent strips, etc.), the reflectance of a materialand optionally the roughness of the surface to be modelled at the sametime on the basis of specular reflections from a planar surface of thismaterial; this planar surface may be curved. These specular reflectionsare sensitive to the properties of the light (shape, intensity,position) and to the properties of the material on which they arerepresented. Constructing a model on the basis of specularities alsoimplies inclusion of the reflectance properties of the material. Theconstruction of this virtual model will allow the specularities to beestimated via projection of the model for unknown viewpoints.

The light sources in question are stationary and are generally not veryfar from the surface presenting the specularities, as is the case forlight sources located in the interior of a building or for an exteriorstreetlight. Nevertheless it will be noted that a source at infinity,such as the sun for example, may also be modelled by the method.

The images with specularities that are processed by the method accordingto the invention are acquired by a sensor the intrinsic parameters ofwhich (focal length and central point) are known.

The situations of the image sensor are considered to be known for eachimage; a 3-D model of the scene is also known, in which only the normalto the plane (or the normals to the planes) on which the specularitiesare located is (are) used. These images may be obtained by a sensormoving along a known path, the images possibly themselves being obtainedin video mode or in manual mode at times that are further apart, or by aplurality of stationary sensors located at various known locations.

The primitives used for the construction of the source model includingthe reflectance of the material correspond to the specular reflectionsin an image. These specularities are caused by the reflection of thelight sources from a reflective (non-Lambertian) surface. It will benoted that the properties of the light source, such as its size, itsposition and its shape, influence the shape of the specularity on asurface. In addition, depending on the reflectance of the material, thesize of these specularity is also modified. Knowledge of these elementsis essential if the shape of the specularity is to be predictable forunknown viewpoints.

In the case of a planar surface, a specularity corresponds to thereflection of a light source with respect to a specular material (aperfectly specular material would correspond to a mirror). This propertyis important because it allows the number of light sources in a scene tobe rapidly identified. In addition, tracking of these specularitiesallows the temporal state of the light sources associated with theseartefacts to be observed.

On the basis of images obtained by a sensor in at least 3 differentpositions, the positions and corresponding orientations of the sensorbeing known, the construction of the model includes the steps shown inthe flowchart of FIG. 1:

A. Detection of Specularities

This step consists in detecting specularities in an image. At the end ofthis step, the resulting image is a gray-scale image. Thesespecularities are detected using the method described in the publicationby the inventors “Generic and real-time detection of specularreflections in images” VISAPP, 2014. This approach is based on theobservation that these specularities stand out more in theHue-Saturation-Value (HSV) color space than, for example, in the RGBspace. Specifically, specularities manifest themselves as high values inthe Value channel and low values in the Saturation channel.

On the basis of this observation, this step is based on automaticthresholding of the Value and Saturation channels depending on theluminosity calculated from the image. In order to maximize the results,preprocessing (=before thresholding) and post-processing (=afterthresholding) are implemented.

Depending on the luminosity calculated from the image, the preprocessingallows difficult images (due to overexposure of the video camera, anunderexposure, or abrupt changes in lighting, etc.) to be anticipated. Acontrast equalization is applied in order to make the specularitiesstand out more in the case of underexposure and in order to limit thesaturation of the image in the case of overexposure. The image resultingfrom this preprocessing allows detection to be performed under betterconditions but also allows images that it will be hard for methods forlocalizing the sensor in a video sequence to exploit to be anticipated.Specifically, the sensor (eye, video camera) is very sensitive tovariations in the light that it receives.

The objective of the post-processing is to filter the various candidatesresulting from the thresholding. After an equalization of the contrast,certain zones in the image remain falsely detected for an overexposedimage in particular. Specifically, a white texture returns a largeamount of light, this possibly leading to false detection, its lightintensity being very high. To respond to this problem, a method forseparating specular spots from ambiguous textures is implemented.

If the image is observed in the Value channel of the HSV, it is easy toobserve, for each specularity, a gradual decrease in the intensity ofthe latter from its center. In order to exploit this criterion, theimage is divided into k-regions of specular candidates. Thissegmentation is carried out using a conventionalbinary-image-segmentation method; for example, the method as describedin the publication by Suzuki and Satoshi “Topological structuralanalysis of digitized binary images by border following” ComputerVision, Graphics, and Image Processing, 30(1): 32-46, 1985 may be used.The maximum circumscribed box is employed for each outline.Specifically, a specularity is nonuniform and is generally veryfragmented. By using a maximum circumscribed box, these fragments areincluded in the calculation. Next, the threshold in the Value channel isgradually modified by one increment of unitary size in each iteration,the resulting variation in the area of these maximum circumscribed boxesbeing observed. If this variation is constant (equivalent to a slightand regular decrease in intensity), this candidate is considered to be aspecularity. If the area decreases suddenly, this candidate probablyrepresents a texture and has therefore been wrongly detected as beingspecular.

B. Tracking of the Specularities

Each specularity is associated with a light source. In order to trackthe state of the light sources in real-time, temporal tracking of eachspecularity is carried out on the basis of the outline of each thereofin successive images, in order to track variation in these outlines. Fortwo successive images, if the preceding and current outline intersect,then the outline is updated and the specularity is tracked. If theoutline disappears with respect to the preceding image, the specularityis no longer tracked (poorly detected outline, light that has beenswitched off). This tracking allows each specularity to be matched inreal-time with a light source reflecting from the specular surface andthat is the origin of this specularity from one image to the next. It isa question so to speak of indexing each specularity with an index thatis specific to the light source that is the cause of said specularity.

In addition, this tracking allows specularities falsely detected in animage to be filtered. Specifically, white textures in the presence ofstrong light return light in large amounts, thereby possibly causing theimage to saturate and leading to false detection of a specularity.However, for a plurality of images, the white texture remains stationaryin the 3-D scene contrary to the specularity which, depending on theviewpoint of the sensor, will move in the scene. The principle of thefiltering according to the invention is based on the dependency of the3-D outline of the specularity, i.e. the outline obtained from the 3-Dmodel of the scene, with respect to the position of the sensor. Thewarping method is used, which consists in calculating, for the specularcandidates, the homography between two images from the situation(=position and orientation) of the sensor for these images. Thecandidate is considered to be a texture if its 3-D outline has not movedin the scene; it is considered to be a specularity if its 3-D outlinehas moved as illustrated in FIG. 3. The dotted outline of the candidatelocated in the center of the book has not moved in the scene between theright-hand image taken at a first viewpoint and the left-hand imagetaken at a second viewpoint and this candidate must be considered as tobe a texture T; the solid-line outline of the candidate in the bottomright-hand corner of the book has moved in the scene between theright-hand image and the left-hand image and it is therefore aspecularity S. To obtain this 3-D outline, each point of a 2-D outlineis considered and ray tracing is carried out in the 3-D model of thescene on the basis of the situation of the sensor of the associatedimage. The intersection between the surface and these rays gives a 3-Doutline that is analyzed for each image.

C. Estimation of Conics Representing the Specularities

According to the applicant, on a planar surface, an entire specularity(i.e. a specularity that is not cut-off) takes the form of an ellipsefor various light sources (lightbulbs, fluorescent strips). Aspecularity more generally takes the form of a conic, but in the rest ofthe description an ellipse will be considered. The basis for thisproperty is the physical model of reflection described by Phong et al.in the publication “Illumination for computer generated pictures”Communications of the ACM, 18(6):311-317, 1975, focusing only on thespecular portion of the Phong equation, which gives the intensity of a3-D point p as:

I _(s)(p)=i _(s) k _(s) ∥{circumflex over (R)}∥∥{circumflex over (V)}∥cos^(n) α,

where {circumflex over (R)} is the normalized direction of a perfectlyreflected ray, {circumflex over (V)} the normalized direction orientedtowards the sensor, n the shininess (reflection coefficient of thelight) of the material for which the ray cuts the plane, k_(s) the ratioof reflection of the intensity of the light, α the angle between R and Vand i_(s) the intensity incident on the surface.

Let us consider a constant angle α and a plane π of specular material.Thus, for this angle, the value of I_(s)(p) is constant. On the basis ofthe Phong equation, it is possible to note that there is a maximumintensity at the point p₀ such that:

p ₀ ∈π,I _(s)(p ₀)=i _(s) k _(s)

α=0

According to Fermat's principle, the path of a light ray between twopoints is always the shortest possible path. The Snell-Descartes law ofreflection may be derived from this principle: the angle θ₁ between thenormal N and the vector L, and the angle θ₂ between the normal N and thevector R are equal. This principle implies that the incident planecontaining N, L and R is orthogonal to π. This principle will be appliedon the basis of two types of sources, point sources and extendedsources.

For a point source such as a light bulb: in the Phong model, for a givenangle α, considering R to be the stationary arm and V (defined by theoptical center P and p a point of the specularity on π) the arm ofvariable length, it is possible to draw a conic on the plane π using aperfect virtual pair of compasses such as described for example in thepublication by J. Van Maanen “Seventeenth century instruments fordrawing conic sections” The Mathematical Gazette, pages 222-230, 1992.

It is interesting to note that the point p₀ is not located at the centerof the ellipse of the specularity. It is possible to consider that thereexists a cut-off angle α_(max) such that:

α_(max),α>α_(max)

I _(s)(p)=0

In addition, this cut-off angle implies that there exists a threshold τsuch that:

∀p∈S,I _(s)(p)>τ, where S is the set of the specular points.

Thus, in the Phong model, the applicant has proved that a point lightsource emits a specularity taking the form of a conic on the plane byusing α_(max), τ and the perfect pair of compasses as illustrated inFIG. 2. Specifically, according to Fermat's principal, the path of alight ray is always the shortest. Thus, the plane formed by the incidentand reflected ray (plane of incidence) is perpendicular to the plane π(reflecting surface) which may be associated with the drawing surface ofthe perfect pair of compasses. If a specularity is entirely visible inthe image on the plane π then the specularity is of elliptical shape.

In the literature, for an extended source such as a fluorescent strip,the extended source is considered to be a discretized segment of pointsources. It is known that these point sources produce specularities ofelliptical shape on a planar surface of a reflective material. If thissegment is considered to be infinitely discretized into point lightsources, then it is possible to represent the specular spots by amaximum circumscribed ellipse. Thus, for an extended light source, thespecularity obtained is also of elliptical shape.

Thus, for each type of light source, for a specular planar surface, itis possible to represent these (whole) specularities as ellipses.

Depending on the quality of the detection, two approaches for estimatingellipses on the basis of a gray-scale image delivered by the detector ofspecularities at the end of step A, are possible:

-   -   Maximum circumscribed ellipse estimation, which allows precise        ellipses to be obtained for detections with little error as        indicated in the publication L. G. Khachiyan, M. J. Todd, “On        the complexity of approximating the maximal inscribed ellipsoid        for a polytope”, Math. Programming 61 (1993)137-159.    -   Fitted ellipse estimation, which allows potential errors due to        the detection to be limited as indicated in the publication        by A. W. Fitzgibbon, R. B. Fisher, et al. “A buyer's guide to        conic fitting”. DAI Research paper, 1996.

D. Choice of Key Images for the Light

For the sake of execution rapidity and precision in the estimation, thesteps of constructing and optimizing the model are not carried out foreach image but only for certain images called “key images”. They arechosen by making a compromise between the quality of the specularitiesdetected in the images and an angle criterion.

It is important to have key images corresponding to viewpointsdistributed over the scene. On the basis of the reflected anglesobtained beforehand, a threshold is set (for example experimentally)between the reflected angle of the current image and each angle of thealready existing key images in order to obtain viewpoints that are farapart such that there is no intersection between the specularitydetected in one key image and the specularity detected in the followingkey image. These key images may be obtained directly (without subsequentsorting) when for example the sensors used are stationary and havealready been localized in locations that are sufficiently far apart tomeet this far-apart-viewpoint criterion.

All these criteria allow optimal viewpoints for the construction of themodel to be obtained. Moreover, the granularity of a specular surfacemay influence the shape of the specularity. Therefore, it may bedifficult to obtain an ellipse corresponding exactly to the specularityas illustrated in FIG. 4 in which it may be observed that the calculatedellipse C corresponds better to the outline of the specularity S in theleft-hand image than in the right-hand image. The quality of thespecularities is estimated by comparing the outline of the specularityobtained from the detector (=from the detection of step A) with theellipse calculated for this specularity. This quality is estimated via apoint/point distance between points on the outline of the specularityand the estimated ellipse. The Jaccard distance may also be used inorder to compare the similarity between the area of the specularity andthe area of the ellipse. These distances are detailed in the optimizingstep.

E. Initialization

In the Phong model, the direction of the point light, and its intensity,must be estimated directly on the basis of a point source. According tothe Phong formula, it is known that a specularity is influenced in sizedepending on the intensity of the light source. In practice, thecalculation of this intensity is not trivial because the sensor is verysensitive to variations in light and the perceived intensity may vary.

It is proposed to reconstruct an empirical model of light sources as avirtual volume. This model includes the intensity and shape of the lightsource and the properties of the materials by reconstructing a quadricon the basis of conics estimated from the specularities associated withthis light source.

For a point source (lightbulb), the model according to the inventiongives an equivalence to this point light source taking the form of asphere as shown in FIG. 5. Specifically, for an omnidirectional pointsource at a fixed position G, there is a point p₀ϵπ such that the anglebetween the normal and the incident ray is equal to the angle betweenthe normal between the reflected ray according to the Snell-Descartesprinciple. In the Phong model, p₀ is a point on the surface of maximumintensity. In FIG. 5, the intersection of the rays P₁q₁, P₂ q₂ and P₃q₃corresponds to G_(s) a point that is the reflection of G with respect toπ; P₁, P₂, P₃ are the various positions of the sensor, q₁, q₂ and q₃being the points of maximum intensity of the corresponding ellipses C₁,C₂, C₃. The point of maximum intensity is not necessarily at the centerof the conic as may be seen in FIG. 8. Returning to FIG. 5, theapplicant has demonstrated that, for each viewpoint, on a planar surfacefor a given specular material, a point source G produces a specularityof elliptical shape. The intersection of the various cones of revolutionof vertices P₁, P₂ and P₃ (optical centers of the sensor) gives a sphereof radius r. This radius r is associated with an intensity and with thereflectance of the given material.

An analogy may be made between the model and a result due to the wavenature of light. Specifically, according to Huygen's principle, lightpropagates in the form of wavelets. A light source is considered to beisotropic, i.e. it emits light waves that propagate in rays in alldirections: at each of the points of space close to the source andreached by the light wave, the state of the electromagnetic fieldoscillates (at the frequency of the source) between a maximum value anda minimum value. At a given time, all the points located at an identicaldistance from the source are in the same state: they are “in phase”. Theenvelope of the points located at the same distance from the sourcemakes a spherical shape in space. The surface of this sphere is calledthe wavefront, which may be considered to be a secondary sourceassociated with the primary source.

It may also be noted that for a given source, there are a plurality ofwavefronts as shown in FIG. 6a in which each circle corresponds to awavefront emitted by the central source in all directions. In thevirtual model, it is possible to make an analogy with wavefronts byconsidering that, for each material, the specularity observed on itssurface will embody a different wave-front level.

For an extended source such as a fluorescent strip:

In computer graphics, it is common to represent an extended source by aset of points (point sources). Thus a fluorescent strip is representedby a set of points distributed over a segment. For these points thewaves, described by a sphere, are independent and originate from eachpoint source. The interference of these spherical waves according to theHuygens-Fresnel principle will give a new wavefront the externalenvelope of which may be generalized by an ellipsoid as illustrated inFIG. 6 b.

Thus, for an extended light source, the model according to the inventionrepresents this source with an ellipsoid by analogy with the shape ofthe wavefront. These observations allow the virtual light-source modelto be generalized using a quadric that may represent a sphere, anellipsoid, a cylinder or even a point.

A few generalities on the construction of a quadric on the basis ofconics will be recalled.

The procedure for initializing a quadric on the basis of conics will bepresented after the formalism of conics and quadrics has been presented.

A quadric is a surface with 9 degrees of freedom which is represented bya symmetric matrix Qϵ

^(4×4) such that any point X on its surface satisfies the equation:

X ^(T) QX=0,

where X is a 3-D point in homogenous coordinates.

The outline occulting a quadric in the image plane, for a given sensorsituation, corresponds to a conic that is an outline with 5 degrees offreedom and that is represented by a symmetric matrix Cϵ

^(3×3) such that any point x on its outline meets the constraint:

x ^(T) Cx=0,

where x is a 2-D point in homogenous coordinates.

In order to express C as a function of Q, the dual space is used insteadof the primal space because the relationship between C and Q is notdirect in the primal space. Using the dual space for the 3-Dreconstruction makes it possible to dispense with depth information. Inthis space, the conics are represented using lines I such that:

I ^(T) C*I=0,

where C* is the dual conic.

For a dual quadric Q*, a representation of the plane π described by thefollowing relationship is used:

π^(T) Q*π=0.

In the dual space, it is possible to use, to within a scale factor, thefollowing relationship:

C*˜PQ*P ^(T),

with Q*=adj (Q) and C*=adj (C), where adj is the adjucate (=thetransposed cofactor matrix).

The method described in the publication Cross et al. “Quadricreconstruction from dual-space geometry” in International Conference onComputer Vision, ICCV, 1998, is used, which allows Q* to bereconstructed by converting the relationship between Q* and C* into alinear system. By vectorising Q, P and C into the form Q_(v), B andC_(v), for n viewpoints with n≥3 we can reconstruct a system by usingspecular reflections taking the form of conics C and the model of thelight source taking the form of Q:

${{Mw} = {{0\mspace{14mu} \mspace{11mu} \begin{pmatrix}B_{1} & {- C_{1,v}^{*}} & 0 & \ldots & 0 \\B_{2} & 0 & {- C_{2,v}^{*}} & \; & 0 \\\vdots & 0 & 0 & \ddots & \vdots \\B_{n} & 0 & 0 & \ldots & {- C_{n,v}^{*}}\end{pmatrix}\begin{pmatrix}Q_{v}^{*} \\\alpha_{1} \\\alpha_{2} \\\vdots \\\alpha_{n}\end{pmatrix}} = 0}},$

With the matrix B_(i) ϵ

^(6×10).

The solution to this system is calculated on the basis of asingular-value decomposition (SVD) of M. It will be noted that α₁corresponds to a scale factor such that:

α_(i) C* _(i,v) =B _(i) Q* _(i,v)

for the viewpoint i.

This procedure is illustrated in FIG. 5 for three viewpoints; the sphereof radius r is constructed on the basis of the three ellipsesrespectively associated with the three specularities. The sphere ofradius r includes the intensity of the light source and properties ofthe material of the surface on which the specularities are present. Thecones of revolution indicated by the dotted lines in the figure andrespectively issued from the points P_(i), are not explicitly calculatedto determine the quadric.

This step is carried out for each indexed light source. Trackingspecularities in the images allows a first model to be constructed foreach light source, independently. At the end of this step, quadrics thateach represent one light source will have been constructed.

F. Optimization

This phase allows the construction of the light-source model to berefined (for each of the light sources) via a minimization, for thevarious viewpoints, of the distance between the projections of thequadric and the outlines of the specularities of each image. Thus, theoptimization is based on the minimization, for all the light sources, ofa non-linear cost function that allows the ellipses generated from theprojections to be made to correspond as best as possible to thecorresponding specularities in the images. This cost function may becalculated differently depending on the chosen distance.

F.1. Jaccard Distance

The Jaccard distance quantifies the similarity between two sets (area ofthe specularity S bounded by a solid line and area of the projectedconic (ellipse in our example) bounded by a dashed line) as shown inFIG. 7b in which each of the two areas and their intersection may beseen, and is given by the equation:

${J\left( {C,S} \right)} = \frac{{C\bigcap S}}{{C\bigcup S}}$

where C is a conic and S a specularity detected in an image.

Jaccard Cost Function:

The objective of this cost function is to maximize, for all theviewpoints, the similarity between the area of the specularity and theconic generated by projecting the quadric.

The cost function is described by the equation:

$\underset{Q_{k}}{Max}{\sum\limits_{i = 1}^{n}\; {J\left( {C_{i,k}^{\prime},S_{i,k}} \right)}}$

where S_(i) are the specular pixels in the image and C′_(i) is the conicfor the image of index i and the light source of index k.

F.2. Point/Point Distance for the Cost Function

A point/conic distance between the outline of a specularity and that ofits associated projected conic is not usable as such. Specifically,calculating the distance between a point and a conic involvescalculating a root of order 4 of a polynomial, this being expensive interms of computing time and not allowing analytical derivatives to beobtained.

The desired solution is to obtain a parameterization of the conic intopoints respectively associated with each outline point, i.e. similar tothe approach described in the publication P. Sturm and P. Gargallo“Conic fitting using the geometric distance” In Asian Conference onComputer Vision, ACCV.2007.

Various cost-function calculations may be used to calculate thispoint/point distance, which is illustrated in FIG. 7a . Two thereof arepresented.

Cost Function with Point/Point Distance

The objective of this cost function is to minimize all the Euclideandistances “d” between the projections C_(i) of the quadric Q for aviewpoint of index i as a function of Q as shown in FIG. 7a . Thisdistance is given by the equation:

$\min\limits_{Q_{k}}{\sum\limits_{i = 1}^{n}\; {\sum\limits_{j = 1}^{m_{i}}\; {d\left( {q_{i,j}^{\prime},q_{i,j,k}} \right)}}}$

where q′_(i,j,k) is the point (represented by a solid circle) of index jof the outline of the specularity S of index i and q_(i,j,k) is thepoint obtained from the parameterization of the conic C (represented bya cross) of index i. This operation is carried out for the light sourcek.

In the case of the cost function for the point/point distance, the conicmust be discretized in each iteration.

Another solution is to use latent variables in order to preserve thesame pairs of points (specularity/conic) instead of recalculating thesepairs in each iteration. However, the constraint to be met is to havelatent variables able to vary only on the outline of its associatedconic. Now, a point of a conic is defined by the relationship: x^(T)Cx=0,

where C is a conic and x a 2-D point in homogenous coordinates.

Thus the cost function may be re-written in the form:

$\left\{ {\begin{matrix}\min\limits_{Q_{k},q_{1,1},\; \ldots \;,q_{n,m_{i}}} & {\sum\limits_{i = 1}^{n}\; {\sum\limits_{j = 1}^{m_{i}}\; {d\left( {q_{i,j,k}^{\prime},q_{i,j,k}} \right)}}} \\{{subject}\mspace{14mu} {to}} & {{q_{i,j,k}^{\prime}C_{j}q_{i,j,k}^{\prime \; T}} = 0}\end{matrix},} \right.$

It will be noted that the quadric to be refined is in generalrepresented by a matrix Qϵ

^(4×4) such that: X^(T)QX=0,

where X is a 3-D point belonging to the surface of the quadric.

This quadric may also be represented parametrically (center of thequadric, scale of each axis, rotation about each axis).

At the end of this step, refined quadrics that each represent onelight-source model will have been constructed. When a plurality ofquadrics coincide, they then represent a model of the same light source.

G. Specularity Prediction:

Once the model has been constructed and refined, it is possible todirectly project this model onto the image plane π_(i) in order topredict the specularity Sp in a current new image acquired using thesensor with a new but known orientation and position P and with the samematerial. This function is particularly useful in the field of augmentedreality. The projection of the (modelled) quadric onto this new imagegives a conic bounding the zone predicted for the specularity.

Although the model is applicable for a given material, it is possible toestimate a specularity, defined to within a factor, for another planarsurface of a specular material different from that of the model but ofknown normal: the center of the projected conic indicates the center ofthe zone predicted for the specularity, the dimensions of thespecularity being those of the conic to within a scale factor. It willalso be noted that for complex objects, by discretization of the outlineocculting the quadric for an unknown viewpoint, it is possible, by raytracing, to predict the outline of the specularity on this object.

Generally, a specularity possesses various intensity levels.Specifically, a specularity possesses a zone of maximum intensity andits intensity decreases with distance from this zone. In our model, thevarious intensity levels are all represented by conics as shown in FIG.8 with 4 conics Cp1, Cp2, Cp3, Cp4 (and therefore 4 intensity levels).It is important to note that the latter are not concentric, as may beseen in this figure. The specularity-predicting method allows themaximum circumscribed outline of the specularity Sp to be estimated onthe basis of the strict projection of the quadric, and the variousintensity levels of the specularity to be estimated by projectingquadrics of various scales, as shown in FIG. 9 for 3 quadric scales. The3 concentric quadrics Q₁, Q₂, Q₃ (illustrating 3 quadric scales)projected onto the image plane π_(i) produce 3 (predicted) conics Cp1,Cp2, Cp3 corresponding to 3 intensity levels of the predictedspecularity Sp. Therefore, our model also allows the zone of maximumintensity of the specularity to be predicted.

H. Correction of the Position and Orientation of the Sensor

When specularity is predicted on the basis of the constructed andoptionally refined model, it is also possible to correct possible errorsin sensor position and orientation on the basis of a calculation of adiscrepancy between a specularity present in the image associated withsaid position and said orientation and a conic resulting from projectingthe modelled quadric onto said image. In practice, this step is carriedout using one of the aforementioned cost functions with the position andorientation of the video camera as minimization parameter; this costfunction is calculated using the outline of the detected specularity andused to construct the model of the source and the outline of thepredicted specularity.

The method has been implemented (detection with temporal tracking ofspecularities, quadric reconstruction and nonlinear refinement) andvalidated quantitatively and quantitatively on synthetic sequences andon real sequences. These tests were carried out in complex environmentswith various light sources (lightbulbs and fluorescent lights) forobjects (examples: table, kitchen, board, book, etc.) made of differentmaterials (steel, plastic, varnished wood, china). The realizedsequences highlight the precision of the method regarding the predictionof specular spots from viewpoints not used for the reconstruction. Inaddition, the video stream was treated in real-time (30 images persecond on average) with the proposed approach.

For all the applications of augmented reality, sensor localization,surface quality control, etc., machine vision using a method accordingto the invention therefore offers an inexpensive, practical andnon-invasive solution.

This method for processing images containing specularities may beimplemented on the basis of images obtained by one movable sensor (or aplurality of stationary sensors) located in a scene, and on the basis ofa 3-D geometric model of this scene.

This method may in particular be implemented using a computer-programproduct, this computer program comprising code instructions allowing thesteps of the method for processing images acquired by the sensor to beperformed. It is then stored on a medium that is readable by computer,such as for example a computer that has access to a 3-D geometric modelof the scene and the positions and orientations of the sensor. Themedium may be electronic, magnetic, optical, electromagnetic or be astorage medium read using infrared light. Such media are for examplesemiconductor memories (random-access memory (RAM) or read-only memory(ROM)), tapes, floppy disks or magnetic or optical (Compact Disk-ReadOnly Memory (CD-ROM), Compact Disk-Read/Write (CD-R/W) or DVD) disks.

Although the invention has been described with reference to particularembodiments, it will be obvious that it is in no way limited thereto andthat it comprises any technical equivalent of the described means andcombinations thereof if the latter fall within the scope of theinvention.

1. A method for processing images of a scene including a surface made ofa material of unknown reflectance, the method comprising the followingsteps: from at least 3 different positions of an image sensor, thepositions and corresponding orientations of the sensor being known,acquiring images of the scene illuminated by a light source, each imagecontaining at least one specularity generated by reflection of the lightsource from said surface and depending on the position of the source, onthe shape of the source and on the intensity of the light source and onthe reflectance of the material; in each image, detecting eachspecularity; and for each specularity, estimating a conic approximatingsaid specularity; comprising the following step: constructing a quadricrepresenting the position of the source, the intensity of the source andthe shape of the light source and the reflectance of the material, onthe basis of the conics, and also of the position and orientation of theimage sensor during the acquisition of the images containing thespecularities respectively approximated by these conics.
 2. Theimage-processing method as claimed in claim 1, the surface having anunknown roughness, the quadric also represents the roughness of thesurface.
 3. The image-processing method as claimed in claim 1, whereinthe scene is illuminated by at least one other light source, each imagecontaining at least one specularity generated by reflection of eachother light source from said surface, and further comprising for eachspecularity: temporal tracking of the specularity and matching saidspecularity, and therefore the conic approximating it, with a lightsource, and wherein the step of constructing a quadric is carried outfor each light source, on the basis of the conics matched with saidlight source, and of the position and orientation of the image sensorsduring the acquisition of the images containing the specularitiesrespectively approximated by these conics.
 4. The image-processingmethod as claimed in claim 1, further comprising the following steps:for each specularity, minimizing the distance between the specularityand a conic resulting from projecting the quadric onto the imagecontaining said specularity along an axis determined by the position andorientation of the sensor, in order to determine a new quadric; anditerating the preceding step until a preset stop criterion is reached,the new quadric resulting from the last iteration being a refinedquadric.
 5. The image-processing method as claimed in claim 1,comprising a step of choosing key images from the acquired images,depending on a predefined criterion of distribution of viewpoints of thesensor in the scene with respect to the light source.
 6. Theimage-processing method as claimed in claim 1, wherein a light source isa point light source and wherein the corresponding quadric is a sphere.7. The image-processing method as claimed in claim 1, wherein a lightsource is an extended source and wherein the corresponding quadric is anellipsoid.
 8. The image-processing method as claimed in claim 1,comprising a step of calculating a prediction of the position and shapeof a conic called the predicted conic approximating a specularity formedon a surface made of a material of known normal, depending on a newposition and a new orientation of said image sensor during theacquisition of a new image and on the projection of the quadric ontosaid new image.
 9. The image-processing method as claimed in claim 8,wherein quadric scales being preset, the conic prediction calculation iscarried out for each scale so as to obtain as many conics as scales,each conic corresponding to an intensity level of the predictedspecularity.
 10. The image-processing method as claimed in claim 8,wherein the material of the predicted specularity is the same as thematerial associated with the quadric, and wherein the size of thepredicted specularity is also predicted.
 11. The image-processing methodas claimed in claim 1, comprising a step of correcting error in theposition and orientation of the image sensor on the basis of acalculation of a discrepancy between a specularity present in the imageassociated with said position and orientation and a conic resulting fromthe projection of the quadric onto said image.
 12. The image-processingmethod as claimed in claim 1, wherein the different positions andorientations of the image sensor are obtained using the same movingimage sensor or using a plurality of stationary image sensors.
 13. Acomputer-program product, said computer program comprising codeinstructions allowing, on the basis of: at least 3 different positionsof an image sensor, the positions and corresponding orientations of thesensor being known, and images of a scene including a surface, made of amaterial of unknown reflectance, illuminated by a light source, whichimages are acquired by said image sensor, each image containing at leastone specularity generated by reflection of the light source from saidsurface and dependent on the position of the source, on the shape of thesource, on the intensity of the light source and on the reflectance ofthe material, the following steps to be carried out when said program isexecuted on a computer: in each image, detecting each specularity; foreach specularity, estimating a conic approximating said specularity; andconstructing a quadric representing the position of the source, theintensity of the source and the shape of the light source and thereflectance of the material, on the basis of the conics, and also of theposition and orientation of the image sensor during the acquisition ofthe images containing the specularities respectively approximated bythese conics.
 14. A computer-program product as claimed in claim 13,wherein the surface having an unknown roughness, the quadric alsorepresents the roughness of the surface.